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Summary  of  March  Bi-Annual  Report 

This  report  summarizes  recent  progress  made  on  the  AVIATR  contract  Task  Order  3,  Condition 
Based  Maintenance  plus  Structural  Integrity  -  Basic  Phase.  Similar  to  previous  progress  reports 
for  this  project  the  organization  of  the  report  follows  the  Work  Breakdown  Structure,  and 
specifically  documents  the  progress  during  the  current  reporting  period,  September  2009  through 
March  20 1 0  on  the  Demonstration  task  (WBS  1.1.2)  which  contains  the  following  key  sub¬ 
sections: 

1.1.2  Demonstration 

1.1. 2.1  ASIP  Integration 

1.1. 2.2  SHM  Analysis 

1.1. 2. 3  Data  Analysis 

1.1. 2.4  Sensitivity  Analysis 

1 . 1 .2.5  Measurement  for  Success 

1 . 1 .2.6  Forward  Looking  Assessment 

This  document  provides  an  update  to  the  progress  on  this  project  during  the  current  reporting 
period.  The  focus  of  the  August  progress  report  was  on  nearly  all  remaining  elements  of  the 
Strategy  Development  task.  The  remaining  elements  of  the  Strategy  Development  task  which 
concluded  in  September  were  presented  at  the  face  to  face  meeting  held  in  December  at  the 
Boeing  facility  in  Seal  Beach,  CA.  A  complete  written  report  of  the  Strategy  Development  will  be 
presented  in  the  final  report  for  this  project. 

Due  to  funding  issues,  the  Demonstration  task  scheduled  to  start  in  October  2009  was  not  fully 
implemented  until  January  2010.  The  level  of  effort  on  the  beginning  tasks  including:  ASIP 
Integration,  SHM  Integration,  and  Data  Analysis  have  been  increased  to  meet  the  original 
schedule.  In  addition,  the  Measurement  for  Success  task  has  been  initiated  much  earlier  than 
originally  planned,  as  it  became  clear  that  there  were  critical  questions  to  be  asked  in  developing 
the  baseline  and  alternative  models  that  would  be  very  beneficial  to  the  detailed  implementation 
of  the  strategy  than  leaving  the  development  of  these  cost  models  toward  the  end  of  the  program. 

The  progress  in  each  of  the  tasks  is  documented  in  detail  in  the  following  sections,  for  simplicity 
the  sections  are  numbered  according  to  the  fourth  level  of  the  WBS  structure  (1.1.2.x,  where  x  is  1 
through  6).  The  Sensitivity  Analysis  and  Forward  Looking  Assessment  tasks  have  not  started  yet 
and  are  scheduled  to  begin  in  August  and  be  completed  in  September. 

The  Demonstration  phase  is  planned  to  be  completed  on  schedule  with  the  delivery  of  the  project 
final  report  at  the  end  of  October.  At  the  present  time,  there  are  no  major  issues  to  report.  The 
project  is  expected  to  be  completed  on  schedule  and  on  budget. 
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1.0  ASIP  Integration 

The  focus  of  this  task  is  to  provide  technical  oversight  of  the  application  of  the  Strategy 
Development  completed  previously  to  the  F-15  FS  626  Bulkhead  structure  demonstration.  While 
providing  direction  and  oversight,  the  key  focus  is  the  coordination  of  the  three  principle  tasks, 
SHM  Integration,  Risk  Analysis  (Data  and  Sensitivity  Analysis  tasks)  and  Measurement  for 
Success  (business  case  development),  while  keeping  a  record  of  indentified  technology  gaps  and 
challenges. 

This  task  runs  the  duration  of  the  Demonstration  Phase,  and  is  key  to  coordinating  the  flow  and 
gathering  of  infonnation  between  the  principle  tasks,  as  well  as  obtaining  the  necessary 
information  from  the  F-15  program  and  maintenance  facilities,  and  our  leverage  projects. 

Two  key  leverage  projects  have  been  identified  and  discussed  from  the  beginning  of  the  project. 
They  are  the  Structural  Health  Monitoring  (SHM)/Assessment  for  Bonded  Repairs  program, 
discussed  briefly  in  Section  2  SHM  Data  Integration,  and  the  F-15  program  which  has  provided 
all  of  the  detailed  FEM  and  crack  growth  analyses,  and  maintenance  infonnation  on  the  bulkhead 
component. 

In  addition  to  the  technical  oversight  and  information  coordination,  the  ASIP  Integration  task  is 
looking  at  how,  so  called,  actual  usage  data,  in  coordination  with  initial  and  updated  design 
usage  spectrum  data  can  be  used  to  develop  a  method  to  estimate,  i.e.  predict,  future  usage.  The 
initial  data  for  this  task  has  been  obtained  from  the  F-15  program.  The  basic  approach  is  to  start 
with  the  initial  assumed  design  usage  data  and  apply  Bayesian  updating  techniques  to  obtain  a 
continuously  evolving  forward  looking  usage  spectrum.  In  the  early  stages  of  service  the 
estimated  usage  will  be  “weighted”  toward  the  design  spectrum.  As  the  aircraft  accumulates 
flight  hours  and  ages  the  weight  in  the  composite  usage  spectrum  will  gradually  shift  toward  the 
demonstrated  usage.  The  assumption  being,  without  other  influences,  that  as  the  aircraft  reaches 
retirement  (here  defined  not  as  the  design  flight  hours  at  retirement,  but  rather  in  a  risk  based 
approach  where  the  cost  of  continuing  to  maintain  the  prescribed  level  of  risk  exceeds  an 
acceptable  cost  value)  it  is  more  likely  that  the  aircraft  will  continue  to  see  service  similar  to 
what  it  has  experienced  in  the  past,  while  still  maintaining  some  portion  of  the  initial,  or  updated, 
design  spectrum.  This  analysis  is  expected  to  be  completed  in  May  and  incorporated  into  the 
Risk/Sensitivity  Analysis,  as  well  as,  Measurement  for  Success  tasks. 
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2.0  SHM  Data  Integration 

The  focus  of  this  task  is  to  provide  SHM  system  information  to  the  CBM+  architecture  under 
evaluation.  To  do  this,  Boeing  is  using  the  Boeing/ AFRL  SHM  Design  Framework  along  with 
the  specific  F-15  626  bulkhead  SHM  system  requirements  developed  previously  for  this  effort. 
Using  the  SHM  Design  Framework  for  selecting  appropriate  SHM  technologies  and  methods 
provides  a  structured  method  for  choosing  appropriate  technologies  to  meet  SHM  data 
requirements.  Candidate  technologies  (addressed  as  system-level  solutions)  are  traded  against 
requirements,  concept  of  operations  and  costs  (when  available).  Tasks  completed  during  this 
reporting  period  focused  on  1)  continuing  to  integrate  and  evolve  the  design  of  the  SHM  system 
with  the  proposed  risk-based  analysis  process  (directly  aiding  the  Data  Analysis  and  Prediction 
tasks),  and  2)  providing  SHM  system  concept  of  operations  (CONOPs)  inputs,  development  cost 
estimates,  and  hardware  cost  estimates  to  support  the  Measurement  of  Success  via  Technical 
Performance  Measurements  (TPMs)  tasks. 

2.1  SHM  System  Design  for  Inclusion  with  the  Risk-Based  Analysis  Process 

The  basic  requirements  for  the  SHM  system  is  that  it  must:  1)  retrofit  existing  vehicles,  2)  be  a 
self  contained  system,  3)  detect  cracks  before  they  are  visible  (i.e.,  before  they  extend  beyond 
structural  buildups  that  require  disassembly  for  visual  detection),  4)  be  relatively  easy  to  install, 
and  5)  be  capable  of  withstanding  all  of  the  operational  environments  the  host  structure  must 
withstand  (e.g.,  debris,  fluids,  temperature  extremes,  etc.).  In  addition,  qualifying  systems  must 
be  capable  of  meeting  or  exceeding  the  minimum  detectible  crack  size  and  Probability  of 
Detection  (POD)  required  by  the  proposed  risk-based  analysis  process.  For  the  two  bulkhead 
crack  locations  (i.e.,  bulkhead  lower  flange  aft  edge  crack  237  location  and  bulkhead  aft  center 
fuselage  FS  629.90  12B  location),  several  monitoring  technologies  were  considered  including: 
Comparative  Vacuum  Monitoring  (CVM),  Smart  Washers,  Crack  Wires,  Piezoelectric  Ultrasonic 
Arrays,  Fiber  Optic  Strain  Sensing,  and  Acoustic  Emission  Systems. 

The  fact  that  the  cracks  are  considered  “hidden”  in  built-up  structure  removes  the  possibility  of 
CVM,  smart  washer  and  crack  wire  solutions.  While  the  total  system  costs  are  being  evaluated  as 
part  of  this  effort  and  are  not  yet  fully  defined,  it  was  assumed  that  systems  that  require  active 
monitoring  during  flight  would  be  cost  prohibitive  (due  to  additional  complexity,  weight, 
qualification  requirements,  and  potential  inclusion  with  existing  on-board  data  systems)  and  thus 
fiber  optic  and  acoustic  emission  options  were  dropped.  The  remaining  technology  capable  of 
meeting  all  of  the  system  level  requirements  is  the  piezoelectric  solution  (note  that  it  must  still  be 
shown  that  the  piezoelectric  solution  does,  in  fact,  meet  the  minimum  detectible  crack  size  and 
POD  requirements). 

A  notional  arrangement  of  piezoelectric  sensors/actuators  to  detect  cracks  at  the  237  and  12B 
locations  is  depicted  in  Figure  1.  Two  initial  options  were  proposed  and  used  in  the  initial  risk 
approach  process  evaluation  for  a  system  that  includes  SHM,  namely: 

1 .  Single  zone:  one  detectable  crack  size  with  probability  of  detection  100%  (or  a  specified 
POD) 

2.  Multiple  zones:  several  detectable  crack  sizes  with  corresponding  probability  of  detection 
for  each  crack  size.  The  largest  one  will  be  the  same  as  the  single  zone  case,  i.e.,  100% 

(or  a  specified  POD). 
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different  POD 


Figure  1.  Boolean  Type  of  SHM  -  Single  Zone  and  Multiple  Zones 


Note  that  the  detailed  SHM  system  design  for  the  items  listed  above  is  being  considered  under 
the  U.S.  Air  Force  Research  Laboratory  Structural  Health  Monitoring  (SHM)/ Assessment  for 
Bonded  Repairs  program  (F336 15-03-2-3300).  With  respect  to  using  the  SHM  Design 
Framework,  critical  design  trade  parameters  include: 

•  Actuation  authority  sensitivity  -  how  to  make  sure  ultrasonic  energy  is  propagating  deep 
enough  into  the  built-up  structure  and  traveling  far  enough  to  be  observed  by  the  sensors 

•  Sensor  topology  -  how  to  optimize  sensor  number  and  placement  to  maximize  sensitivity 
to  ultrasonic  energy  scattered  or  reflected  by  the  flaw/damage  of  interest 

•  Input  signal  source  optimization  -  how  to  determine  the  correct  waveform(s)  to  input  to 
the  structure  that  will  be  most  sensitive  to  flaws/damage  and  detectible  by  the  sensors 

•  Algorithm  development  -  how  to  process  the  sensor  data  to  produce  the  most  accurate 
and  reliable  flaw/damage  size  estimates 

This  AVIATR  CBM+SI  effort  is  complimenting  that  effort  (and  vice  versa)  by  working  to 
address  the  specific  minimum  detectible  crack  size  and  POD  requirements  (described  in  Section 
3),  as  well  as  developing  the  proper  methods  for  modeling,  estimating  and  quantifying  SHM 
derived  PODs  and  their  respective  reliabilities.  As  the  SHM  system  design  matures,  so  will  the 
PODs  for  use  in  the  risk  assessment  studies. 

2.2  SHM  System  CONOPs,  Costs  and  Contribution  to  TPMs 

During  the  current  reporting  period,  SHM  system  information  was  collected  to  support  the 
definition  of  concept  of  operations,  development  cost  estimates,  and  hardware  cost  estimates  to 
support  the  Measurement  of  Success  via  Technical  Performance  Measurements  (TPMs)  tasks. 
This  information  is  summarized  in  Section  3.2  “Perform  Analysis  to  Model  Identified 
Uncertainties”  and  Section  5  “Measurement  of  Success  via  Technical  Performance 
Measurements  (TPM)”. 
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2.3  Next  Steps 

The  remainder  of  the  SHM  Integration  effort  will  focus  on  methods  for  estimating  the 
perfonnance  of  the  proposed  SHM  system,  specifically  on  estimating  PODs  for  the  237  and  12B 
locations.  Doing  this  will  require  developing  models  to  relate  the  actual  flaw  size  to  the 
predicted  flaw  size,  understanding  and  expressing  this  information  with  respect  to  critical  flaw 
sizes,  and  finally  developing  estimated  POD  functions  to  be  used  in  refined  risk  assessment 
evaluations.  In  addition,  this  part  of  the  overall  effort  will  continue  to  refine  the  approach  for 
integrating  the  SHM  data  into  the  CBM+  architecture  based  on  the  CBM+SI  requirements  as 
well  as  how  the  data  are  provided  to  the  overall  maintenance  process. 
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3.0  Data  Analysis  and  Prediction 

The  objective  of  this  task  is  to  apply  the  developed  strategy  and  develop  risk  assessment 
approaches  (tasks  1.1. 2.3  and  1. 1.2.4)  integrated  into  a  CBM+SI  framework.  The  premise  is  that 
formulating  a  risk  based  approach  will  provide  benefits  by  reducing  the  overall  maintenance  cost 
and  improving  the  aircraft’s  availability.  The  approaches  will  be  demonstrated  by  solving  the 
proposed  beta  test  cases. 

Through  the  application  of  the  developed  strategy  in  this  demonstration,  it  is  expected  that  the 
strategy  will  be  updated  and  improved  as  relevant  information  and  issues  arise.  Specifically, 
through  the  Demonstration,  key  elements  for  the  Strategy  Development  will  be  continuously  re¬ 
evaluated.  From  this  demonstration  phase  experience,  continue  to  improve  the  proposed 
CBM+SI  strategy: 

•  Assess  that  the  requirements  are  sufficient 

•  Assess  that  the  architecture  prototype  is  suitable 

•  Assess  that  the  proposed  data  flow  is  accurate  and  easily  applied 

•  Remove  technology  gaps  and  shortcomings  from  the  experience  gained,  as  appropriate, 
and  identify  and  document  others  which  are  outside  the  scope  of  this  project. 

3. 1  Collect  Data  Required  for  ASIP  Reliability  and  Risk  Analysis 

The  first  step  in  this  task  was  to  detennine  the  required  data  based  on  the  Strategy  Development 
tasks,  Requirements  (WBS  1 . 1 . 1 . 1)  and  Architecture  Prototype  (WBS  1 . 1 . 1 .2)  for  ASIP 
reliability  and  risk  analysis. 

The  data  required  for  ASIP  reliability  and  risk  analysis  are:  on-board  SHM  data,  Non-destructive 
Inspection  (NDI)  data,  detenninistic  crack  growth  analyses  data,  full-scale  testing  results, 
maintenance  experience  or  repair  quality,  material  strength,  individual  aircraft  tracking  data  (i.e., 
load  exceedance  data),  initial  quality  flaw,  failure  report,  life  cycle  cost  data,  and  maintenance 
hours  before  and  after  using  the  proposed  information  tools  (e.g.,  IETM),  etc. 

The  data  collection  was  coordinated  with  the  following  parallel  tasks,  ASIP  Integration,  SHM 
Integration,  and  TPM  Measurement  of  Success. 

Data  have  been  collected  for  two  F-15  Bulkhead  critical  locations,  12B  (cracks  emanating  from  a 
series  of  18  bolt  holes)  and  237  (cracks  which  develop  in  the  built-up  interface  section  of  the 
lower  flange  on  the  bulkhead).  How  these  data  are  being  used  to  model  the  constituent  random 
variables  in  the  reliability  and  risk  analysis  will  be  discussed  in  the  following  subsections. 

3.2  Perform  Analysis  to  Model  Identified  Uncertainties 

In  the  Strategy  Development  phase  the  Architecture  Prototype  task  evaluated  the  current  ASIP  risk 
assessment  process  to  determine  the  requirements  for  modeling  uncertainty,  and  that  analysis  is 
implemented  in  this  section,  with  the  exception  of  SHM  data  which  is  discussed  below. 

Details  on  how  to  model  the  collected  uncertainty  data  into  random  variables  has  been 
summarized  in  the  August  2009  progress  report.  In  addition  the  requirements  for  structural 
health  monitoring  data  requirements  was  also  detailed  in  that  report,  and  referenced  here. 

Based  on  the  above,  a  summary  of  all  input  data,  excluding  SHM  data  which  is  discussed 
below,  for  the  Bulkhead  component’s  12B  location  can  be  seen  in  the  Appendix  A  at  the 
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end  of  this  report.  Uncertainty  modeling  associated  with  SHM  data  is  coordinated  with  the 
SHM  Integration  task  considering  two  proposed  approaches  of  a  single  or  multiple  zone 
SHM  probability  of  detection  (POD)  capability. 

As  discussed  in  the  Strategy  Development:  Requirements  task,  two  bookends  of  SHM  capability 
have  been  proposed.  First,  at  the  limited  capability  end  a  Boolean  approach,  this  would  give 
limited  binomial  crack  detection  infonnation  within  the  monitoring  area  and  second,  a  high 
capability  approach  that  would  be  able  to  detennine  crack  length  and  orientation  with  high 
confidence.  However,  with  project  constraints,  only  the  SHM  -  modified  Boolean  approach 
considering  single  and  multiple  zones  will  be  developed  and  demonstrated  as  shown  in  Figure  1. 
The  SHM  -  high  fidelity  system  will  not  be  able  to  be  developed. 

For  the  Boolean  type  of  SHM,  we  have  the  following  definitions: 

1 .  Single  zone:  one  detectable  crack  size  with  probability  of  detection  100%  (or  a 
specified  POD,  and  confidence) 

2.  Multiple  zones:  several  detectable  crack  size  zones  with  corresponding  probability  of 
detection  for  each  zone.  The  largest  one  will  be  the  same  as  the  single  zone  case,  i.e., 
100%  (or  a  specified  POD,  and  confidence).  For  the  other  crack  size  zones,  the 
probability  of  detection  may  be  different. 

Based  on  the  above  definitions,  crack  size  criteria  data  were  determined  based  on  the  F-15 
program  requirements  and  experience  for  the  two  locations  under  consideration  ( 12B,  bolt  holes, 
and  237,  built-up  section)  at  the  FS  626  Bulkhead  area: 

1.  For  12B  (boltholes): 

a.  0.03  inches  -  easy  to  repair  by  drilling  out  the  hole  to  the  next  oversize  level. 
When  crack  is  smaller  than  0.03”,  the  maintenance  authority  is  instructed  to 
use  the  next  oversize  fastener. 

b.  0.05  inches  -  Bolt  hole  eddy  current  NDI’s  capability.  When  crack  is  smaller 
than  0.06”  but  larger  than  0.03”,  the  maintenance  authority  is  instructed  to  use 
the  Force-Tec  system.  A  Force-Tec  is  a  thick  walled  bushing,  cold  worked 
into  place. 

c.  0.12  inches  -  this  crack  size  is  still  repairable  by  using  Force-Tec  technology. 
The  actual  repairable  size  is  up  to  0. 18  inches  (double  the  size  of  the  holes). 
When  the  crack  is  between  0.06  in.  and  0. 18  in.,  engineering  may  instruct 
maintenance  personnel  to  "pull"  the  hole  off  center  to  help  cover  the  crack.  An 
issue  is  that  the  manuals  do  not  allow  the  typical  maintenance  crew  to  do  this 
and  additional  engineering  support  is  required. 

d.  0.18-  critical  crack  length.  When  the  crack  is  larger  than  0. 1 8”,  a  replacement 
of  the  BH  is  the  only  solution  currently  available.  A  potential  repair  strategy  is 
currently  in  development  stage  by  the  F-15  program  for  this  large  crack  size. 
However,  the  cost  of  repair  for  the  large  crack  like  this  is  expected  to  be  much 
more  costly  than  the  previous  three  stages. 

2.  For  237  (built-up  section  along  flange) 

a.  0.03  inches  -  easy  to  repair  by  blending 

b.  0.05  inches  -  Limit  of  ultrasonic  NDI  inspection  capability  for  fillet  radii 
inspection 
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c.  0.1-  considered  repairable  by  F-15  program;  repair  technique  in  development. 

d.  0.3  -  crack  growth  analysis  critical  crack  length 

3.  Note  that  for  the  bolt  hold  location  (12B)  235.6  hours  are  required  to  grow  the  crack 
from  0.12  inches  to  0.18  inches.  Considering  the  built-up  flange  section  (237)  1429 
hours  is  required  to  grow  the  crack  from  0.1  inches  to  the  critical  crack  size  of  0.3 
inches.  Therefore,  the  size  for  the  single  zone  Boolean  SHM  is  0. 12  inches  and  0. 1 
inches  for  12B  and  237  locations  respectively.  Their  corresponding  probability  of 
detection  will  be  detennined  as  part  of  the  SHM  Integration  task  and  documented  in  a 
subsequent  report. 

The  other  key  data  required  is  how  much  time  is  required  for  the  repair  process.  Based  on  the 
above  definitions  the  following  information  was  provided  by  the  F-15  program: 

Zone  12B:  18  holes  (7-8  incidents  reported) 

1 .  Assume  plane  is  open  and  inspected,  and  crack  was  found 

a.  When  crack  is  smaller  than  0.03”  -  30  minutes  to  repair 

b.  When  crack  is  smaller  than  0.06”  but  larger  than  0.03”  -  30  minutes  to  repair 

c.  When  crack  is  smaller  than  0.18”  but  larger  than  0.06”  -1.5  hours  to  repair 

d.  When  crack  is  larger  than  0.18”-  replace  bulkhead 

i.  The  F-15  program  is  working  on  an  alternate  repair  approach.  The 
approach  is  expected  to  be  completed  in  July,  and  the  program 
anticipates  a  1000  -  1500  labor  hours  repair  time  frame. 

Zone  237:  2  fillet  radius  (2  incidents  reported) 

1 .  Assume  plane  is  open  and  inspected,  and  crack  was  found 

a.  When  crack  is  smaller  than  0.03”-  30-60  minutes  to  repair 

b.  When  crack  is  smaller  than  0.1”  but  larger  than  0.03”  -  30-60  minutes  to 
repair 

c.  When  crack  is  smaller  than  0.3”  but  larger  than  0.1”-  possible  plane  killer 

i.  The  F-15  program  is  working  on  a  repair  approach.  The  approach  is 
expected  to  be  in  the  1000  -  1500  labor  hours  repair  time  frame. 

d.  When  crack  is  larger  than  0.3”  -  replace  bulkhead 

In  addition  to  the  crack  size  for  detection,  it  is  important  to  understand  the  data  collection  time 
requirements  for  both  NDI  and  SHM  approaches. 

1.  For  NDI  inspection,  to  access  the  bulkhead  and  perfonn  the  inspection,  it  takes 
around  2-3  hours,  based  on  information  from  F-15  program  maintenance  records. 

2.  For  SHM  inspection,  since  this  is  a  retro-fit  application,  all  data  acquisition  hardware 
and  software  will  be  off-board  in  order  to  mitigate  any  weight  or  system  penalties  - 
weight  neutral  target.  The  information  provided  below  is  based  on  experience  from 
the  SHM  community  of  experts. 

a.  Data  collection  will  involve  bringing  a  portable  system  to  the  aircraft 
consisting  of  a  computer,  data  acquisition  hardware  and  power  supply. 
Options  to  this  include  adapting  a  system  to  fit  in/on  an  existing  maintenance 
"rack"  or  system  and  to  have  external  power  supplied.  The  data  collection 
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system  will  provide  power  to  the  onboard  sensors  and  collect  the  data  from 
the  sensors.  In  general,  a  "source"  actuator  will  introduce  ultrasonic  energy 
into  the  structure  and  one  or  more  sensors  will  pick  up  the  transmitted  energy. 
The  data  acquisition  hardware  will  collect  the  data  picked  up  by  the  sensors 
and  will  store  it.  Once  connected  and  powered  up,  the  actual  collection  of 
data  should  take  no  more  than  5  or  10  minutes  and  is  completely  automated 
(most  likely  an  executable  with  a  "start"  button  on  it). 

b.  Data  processing  takes  the  stored  data  from  the  sensors  and  turns  it  into  useful 
information.  Based  on  the  capability  desired,  the  information  will  be  Boolean 
(i.e.,  "exceeds"  or  "does  not  exceed"  for  a  specific  zone)  or  detailed  crack 
information  (crack  specifics  like  orientation,  length,  etc.). 

c.  Data  transfer  is  the  process  of  moving  the  crack  information  to  a 
location/computer  that  will  be  performing  the  CBM+SI  analysis. 
Theoretically,  this  could  happen  in  several  different  ways  ranging  from  data 
entered  in  a  maintenance  log  by  the  technician  who  ran  the  data  acquisition 
process  and  data  collection  routine  to  an  uploaded  data  file  (via  thumb  drive, 
or  Ethernet,  etc.)  sent  back  to  cognizant  engineers  for  analysis. 

The  proposed  risk  assessment  analysis  must  be  able  to  integrate  the  SHM  information  and  its 
related  uncertainties.  In  addition,  based  on  the  inspection  and  repair  data  collected,  it  is 
important  to  detennine  the  optimal  maintenance  schedule  using  the  risk-based  approach  with 
consideration  of  both  NDI  and  SHM.  For  the  purposes  of  this  report  and  to  facilitate  the 
demonstration  of  the  incorporation  of  SHM  data  into  the  approach,  the  assumption  is  made  that 
the  SHM  system  has  the  same  capability  as  that  of  existing  NDI  technology.  The  SHM 
Integration  task  will  develop  a  POD  data  set  to  be  used  in  future  analysis  and  business  case 
assessments.  Based  on  this  assumption,  it  is  possible  to  perfonn  current  risk  assessment  analysis 
process,  an  updated  analysis  process  with  the  addition  of  SHM,  and  then  a  risk-based 
optimization  process  and  code  to  identify  the  optimal  maintenance  schedule  with  considering  of 
both  NDI  and  SHM.  These  preliminary  and  exploratory  results  will  be  shown  in  the  following 
sections. 

3.3  Implement  Proposed  Risk-based  Methodology  And  Develop  Code 

Step  1 .  Based  on  Strategy  Development  proposed  data  flow  (subtask  1.1.1 .3),  implement 
proposed  risk-based  methodology  and  develop  code  to  incorporate  SHM  data  obtained  from  task 
1.1. 2. 2  (SHM) 

The  purpose  of  this  task  is  to  implement  the  proposed  risk-based  methodology  and  develop  the 
necessary  code  to  calculate  the  risk  with  consideration  of  SHM  data  and  to  detennine  logistic  or 
maintenance  action  needs.  In  the  following,  the  methodology  part  has  been  defined  and 
discussed;  however,  the  code  portion  will  not  be  fully  developed  until  the  late  April  or  early  May 
timeframe. 

The  proposed  risk-based  analysis  process  with  addition  of  SHM  is  discussed  below. 

1 .  With  the  addition  of  SHM,  the  current  risk  assessment  process  will  not  be  changed  but  it 
may  increase  the  analysis  iterations  as  shown  in  Figure  2.  When  there  are  no  cracks  found 
by  SHM  before  the  end  of  selected  inspection  interval,  which  shall  be  determined  by  using 


9 

Approved  for  public  release;  distribution  unlimited. 


the  original  risk  assessment  process  based  on  the  1  .E-7  risk  requirement  without 
considering  the  SHM  capability,  an  NDI  inspection  will  be  perfonned  as  usual. 

However,  when  a  crack  is  found  before  the  end  of  inspection  interval,  it  is  important  to 
determine  the  impact  of  SHM.  To  perfonn  the  above  analysis,  the  following  information 
shall  be  required: 

a.  Input  data  requirements  (1.  Aircraft  usage  characterization,  2.  Crack  growth  and 
residual  strength  based  on  demonstrated  usage  and  location  of  interest,  material 
parameters,  and  stress  intensity  solution  (a  =  K/a),  3.  NDI  inspection  data) 

b.  Input  data  requirements:  SHM  data 

i.  Boolean  type  of  model  with  single  or  multiple  zone  data  shall  be  available. 

c.  Input  data  uncertainties  modeling 

d.  Risk  requirement  from  MIL-STD-1530C 

e.  Computation  tool:  An  existing  computational  tool  will  be  used  with  stand  alone 
augmented  code  to  accommodate  the  additional  SHM  capability  to  solve  the  risk 
assessment  problem,  calculate  the  percentage  of  crack  detected  and  repaired,  and 
identify  the  optimal  maintenance  schedule. 


**  At  the  end  of  ith  or  the  last  usage  interval,  a  NDI  can  be  applied  to  detection  the  crack  (as  an  option), 
this  value  was  determined  by  the  original  RBDMS  code  without  considering  the  SHM. 


Figure  2.  Risk  Assessment  Engineering  Analysis  Process  with  SHM  Addition 

Given  the  SHM  Boolean  capability  plus  the  off-board  data  extracting  design,  it  is  impossible  to 
monitor  the  crack  all  the  time.  Instead,  the  SHM  should  be  perfonned  at  the  pre-selected  time 
intervals.  In  the  following,  the  analysis  procedure  for  SHM -based  risk  assessment  process  will 
be  discussed  in  details: 
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1 .  With  the  addition  of  SHM,  the  current  risk  assessment  process  will  not  be  changed 
drastically  but  it  may  increase  the  analysis  iterations.  Boeing’s  Risk-Based  Design  and 
Maintenance  System  (RBDMS)  code  will  be  used  and  integrated  with  additional  SHM 
capabilities  to  perform  the  analysis.  In  general,  data  required  for  the  analysis  are  essential 
the  same  except  for  the  addition  of  SHM  data.  Based  on  this  concept,  the  following 
analysis  process  was  defined  especially  for  the  methodology  to  evaluate  the  POD  for 
SHM  and  how  to  update  the  crack  size  distribution  after  performing  a  SHM  inspection 
and  repair. 

a.  Perform  the  crack  growth  analysis  up  to  the  inspection  time. 

b.  Calculate  the  risk  based  on  the  same  risk  assessment  process. 

c.  Calculate  the  percentage  of  cracks  detected  and  repaired  based  on  the  SHM’s 
defined  capability. 

i.  For  the  SHM  single-zone  Boolean  case: 

•  Definitions  for  SHM  single-zone  Boolean  case: 

a.  aR  =  selected  crack  size  for  SHM  singe-zone  Boolean 
detection  and  repair.  Note  that  crack  sizes  less  than  aR  will 
be  assumed  not  to  be  detected.  (For  this  discussion  a 
binomial  variable  is  used,  in  subsequent  detailed  analysis 
presented  later  in  this  report  a  continuous  distribution  is 
assumed.) 

b.  PExceed  =  crack  size  distribution  with  a  >  aR  =  Pr(a  >  aR) 

c.  PD  =  Probability  of  detection  for  a  >  aR  =  SHM  capability 

•  %  of  cracks  detected  and  repaired  =  PR  =  PD  x  PExceed 

•  %  of  cracks  detected  and  not  repaired  (with  crack  size  larger  than 
aR)  =  (1-PD)  x  PExceed 

ii.  For  SHM  multiple-zones  Boolean  case: 

•  Definitions  for  SHM  multiple-zones  Boolean  case: 

a.  P exceed  1  =  Pr(a  >  al) 

b.  Pexceed2  =  Pr(a  >  a2) 

c.  Pexceed3  =  Pr(a  >  a3) 

d.  Note  that  al  <  a2  <  a3 

e.  Pexceedl  >  Pexceed2  >  Pexceed3 

f.  Probability  of  detection  should  have  the  following 
condition:  PD1  <  PD2  <  PD3.  These  data  should  be 
defined  based  on  SHM  capability. 

•  %  of  cracks  detected  and  repaired  =  PR  =  [0.  x  PExceed  1]  +  PD1  x 
(PExceed  l-PExceed2)  +  PD2  x  (PExceed2  -  PExceed3)  +  PD3  x 
PExceed3 

•  %  of  cracks  detected  and  not  repaired  =  (1  -  PD1)  *  (PExceed  1  - 
PExceed2)  +  (1  -  PD2)  x  (PExceed2  -  PExceed3)  +  (1  -  PD3)  x 
PExceed3 
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•  %  of  cracks  detected  and  not  repaired  with  crack  size  larger  than 
a3  =  (1  -  PD3)  x  PExceed3 

d.  Update  the  crack  size  distribution  by  replacing  %  of  cracks  detected  (=  PR)  with  the 
repaired  crack  size  distribution  (usually  considered  equal  to  an  initial  flaw  size 
distribution,  i.e.,  as  good  as  a  new  one)  and  merge  with  the  original  crack  size 
distribution  which  has  not  been  repaired. 

i.  For  the  SHM  single-zone  Boolean  case: 

•  Based  on  the  above  definitions  for  the  SHM  single-zone  Boolean  case, 
the  updated  crack  size  distribution  =  ( 1  -  PExceed)  x  (crack  size 
distribution  with  a  <  aR  -  a  truncated  distribution)  +  PExceed  x  [  { ( 1  - 
PR)  x  (crack  size  distribution  with  a  >  aR  -  a  truncated  distribution)} 

+  {(PR)  x  (repaired  crack  size  distribution)]. 

ii.  For  the  SHM  multiple-zones  Boolean  case,  the  process  may  be  more 
complicated  than  the  single-zone  case  and  a  numerical  process  is  required  to 
establish  the  updated  crack  size  distribution. 

•  Based  on  the  above  definitions  for  the  SHM  multiple-zones 
Boolean  case,  the  updated  crack  size  distribution  =  (1  -  Pexceedl) 

x  (crack  size  distribution  with  a  <  al  -  a  truncated  distribution)  + 
(P exceed  1  -  Pexceed2)  x  [( 1  -  (P exceed  1  -Pexceed2)  x  PD  1 )  x 
(crack  size  distribution  with  al<a  <a2-a  two-sided  truncated 
distribution)  +  (Pexceedl -Pexceed2)x  PD1  x  (repair  crack  size 
distribution)]  +  (Pexceed2  -  Pexceed3)  x  [(1-  (Pexceed2- 
Pexceed3)  x  PD2)  x  (crack  size  distribution  with  a2  <  a  <  a3  -  a 
two-sided  truncated  distribution)  +  (Pexceed2-Pexceed3)  x  PD2  x 
(repair  crack  size  distribution)]  +  (Pexceed3)  x  [(1-  (Pexceed3)  x 
PD3)  x  (crack  size  distribution  with  a  >  a3  -  a  one-sided  truncated 
distribution)  +  Pexceed3  x  PD3  x  (repaired  crack  size 
distribution)]. 

e.  Based  on  the  updated  crack  size  distribution,  calculate  the  updated  risk. 

2.  Based  on  the  updated  crack  size  distribution,  perform  crack  growth  to  the  next  SHM 
inspection  interval  and  repeat  the  above  process  from  steps  1  .a  to  1  .e.  The  process  will 
then  be  repeated  until  the  selected  number  of  inspections  has  been  reached.  Note  that  this 
process  should  have  an  option  to  perform  NDI  POD  inspection  as  well  as  the  SHM 
inspection. 


The  above  developed  SHM-related  strategy  will  be  coded  and  completed  in  early  May  timeframe 
and  an  updated  risk-based  analysis  code  will  be  developed. 


Step  2.  Review  identified  technology  shortcomings  and  gaps  and  develop  necessary  strategy  to 
resolve  these  shortcomings  and  gaps. 

This  step  will  be  complete  after  a  thorough  demonstration  case  has  been  solved.  The  following 
are  extracted  from  the  previous  progress  report  to  reiterate  the  identified  shortcomings  and  gaps. 
It  is  anticipated  that  some  of  these  may  be  resolved  in  the  execution  of  this  phase  of  the  project. 
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1 .  For  the  proposed  CBM+SI  architecture,  advanced  risk  assessment  analysis  strategy 
was  used  to  deal  with  the  uncertainties  incurred  from  the  complex  crack  growth 
model,  loads,  material  strength,  NDI  and  SHM  data.  It  is  important  to  have  a  robust 
strategy  that  will  satisfy  Air  Force’s  requirements  and  identify  the  optimal 
maintenance  schedule  to  save  cost  and  improve  aircraft’s  availability.  The  following 
questions  come  from  the  risk  assessment  analysis  processes  and  tools  development. 
Some  resolutions  or  discussions  are  included  for  reference.  Additional  research  and 
development  to  answer  all  of  these  questions  are  needed. 


a.  Flow  to  update  the  crack  size  distribution? 

b.  How  to  determine  the  next  inspection  interval? 


c.  How  to  resolve  the  missed  crack  issue  which  deterministic  approach  has  the  most 


difficulty?  This  issue  must  be  studied  in  detail  and  documented. 

3.4  Perform  CBM+SI  Risk  Analysis  On  Selected  Beta  Testing  Cases 


Step  1 .  Based  on  the  collected  data  (1.1 .2.3.1),  modeled  variables  (1.1 .2.3.2),  proposed  method 
and  developed  code  (1.1. 2. 3. 3),  perform  a  CBM+SI  risk  assessment  for  the  selected  beta  testing 
cases  (1.1. 1.4). 


Perfonn  a  CBM+SI  risk  analysis  based  on  the  data  acquired  and  calculate  the  risk  that  shall  be 
used  to  compare  with  the  ASIP’s  risk  requirements  in  order  to  determine  the  optimal 
maintenance  schedule.  Since  SHM’s  Boolean  capabilities  are  currently  in  development,  it  is 
necessary  to  consider  the  following  four  maintenance  options  for  the  selected  BH  component: 

1 .  Deterministic  based  approach:  By  calculating  the  corresponding  risk  based  on  the 
inspection  intervals  determined  by  using  the  deterministic  approach,  it  is  possible  to 
compare  with  other  options.  In  the  past,  the  deterministic  approach  was  based  on  the 
safety  factor  (=  2)  but  the  actual  risk  remains  unknown. 

2.  Risk  based  ( 1  .E-7  requirement)  with  NDI  inspection:  This  approach  used  the  MIL- 
STD-1530C  risk  requirement  of  a  Single  Flight  Probability  of  Failure  (SFPOF)  not  to 
exceed  1  .E-7  to  determine  the  inspection  schedule.  At  the  time  of  inspection,  NDI 
will  be  performed  and  %  of  cracks  detected  and  repaired  will  be  estimated. 

3.  SHM  based  from  time  zero:  This  approach  uses  the  key  advantage  of  SHM  of  easy 
access  to  inspect  the  component.  Therefore,  a  preset  time  interval  is  used  to 
detennine  the  risk  and  perfonn  inspection  and  repair,  i.e,  update  the  crack  size 
distribution  in  the  analysis.  Potentially,  a  large  number  of  inspections  will  be 
performed  but  as  a  result  the  risk  will  be  reduced  to  the  minimum. 

4.  Risk  based  for  the  first  inspection  and  then  SHM  based:  This  approach  is  a 
combination  of  approaches  2  and  3.  Based  on  approach  2’s  first  inspection  time 
determined  by  a  SFPOF  of  1  .E-7,  it  will  reduce  the  number  of  inspections  during  the 
initial  phase  of  crack  growth.  After  the  first  inspection,  the  remaining  inspections  are 
based  on  a  pre-selected  SHM  inspection  time  interval. 


13 

Approved  for  public  release;  distribution  unlimited. 


Based  on  the  data  shown  in  Appendix  A,  the  component  12B  will  be  solved  using  the  above  four 
approaches  and  their  results  are  shown  as  follow: 

3.4.1  Deterministic  Based  Approach  Results: 

Since  the  damage  tolerance  life  for  the  F-15  Bulkhead  12B  location  is  equal  to  1300  hours,  the 
inspection  interval  becomes  1300/2  =  650  hours  with  consideration  of  a  safety  factor  of  2.  For 
the  follow-on  inspection  interval,  because  the  minimum  identifiable  crack  size  by  NDI  is  equal  to 
0.05”,  the  second  (and  follow-on)  inspection  interval  was  found  equal  to  650  hours  with 
consideration  of  a  safety  factor  of  2.  Based  on  the  same  650  hours  inspection  interval,  a  total  of 
1 8  inspections  were  considered. 

The  purpose  of  running  this  option  is  to  check  if  the  proposed  inspection  interval  (650  hours) 
will  accurately  protect  the  safety  of  aircraft,  i.e.,  a  risk  that  less  than  1  .E-7.  The  SFPOF  for  the 
deterministic  approach  is  plotted  in  Figure  3.  The  calculated  SFPOF  exceeded  the  risk 
threshold  of  l.E-7  at  4550hours. 

1.0E-05 

1.0E-06 

1 .0E-07 

^  1.0E-08 

°  1.0E-09 

■§)  1.0E-10 

*  1.0E-11 

c  1.0E-12 
55 

1.0E-13 
1.0E-14 
1.0E-15 
1.0E-16 

~ooooooo 
o  o  o  o  o  o  o 

CM  ^  <D  CO  O  CM 

Hours 

Figure  3.  Single  Flight  Probability  of  Failure  -  Deterministic  Approach 


The  percentage  of  cracks  detected  and  repaired  with  each  inspection,  i.e.,  the  portion  of  the 
original  crack  size  distribution  that  is  replaced  with  the  repaired  crack  size  distribution  is  shown  in 
Figure  4.  In  addition,  in  these  analyses,  the  POD  is  a  continuous  distribution  and  not  the  binomial 
value  considered  earlier. 
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1.4 


Inspection  Interval 

Figure  4.  Percentage  of  Cracks  Detected  -  Deterministic  Approach 

3.4.2  Risk-based  NDI  Approach  Results: 

This  approach  was  based  on  the  MIL-STD-1530C  risk  requirement  of  l.E-7  to  determine  the 
inspection  schedule.  At  the  time  of  inspection,  for  this  case,  NDI  will  be  performed  and  %  of 
crack  will  be  detected  and  repaired.  The  same  strategy  can  be  applied  to  apply  SHM 
especially  for  the  first  inspection  interval. 

In  the  following,  both  the  SFPOF  and  the  percentage  of  crack  detected  are  plotted  in  Figure  5  and 
6,  respectively. 


Figure  5.  Single  Flight  Probability  of  Failure  -  Risk-based  NDI  Approach 
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Figure  6.  Percentage  of  Cracks  Detected  -  Risk-based  NDI  Approach 

3.4.3  SHM  Based  From  Time  Zero  Approach  Results: 

This  approach  uses  the  key  advantage  of  SHM  in  easy  access  to  inspect  the  component  so 
frequent  inspection  and  repair  can  be  done.  However,  due  to  current  stand  alone  off-board  SHM 
data  analysis  capability  to  monitor  the  component  where  all  the  time  on  functionality  is  not 
possible,  a  preset  time  interval  is  used  to  perfonn  inspection  and  repair  and  at  the  same  time 
calculating  the  corresponding  risk.  For  this  approach,  potentially,  a  large  number  of  inspections 
are  required  but  the  risk  should  be  reduced  to  the  minimum. 


In  the  following,  both  the  SFPOF  and  the  percentage  of  crack  detected  are  plotted  in  Figure  7  and 
8,  respectively. 
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Figure  7.  Single  Flight  Probability  of  Failure  -  SHM  from  Time  Zero  Approach 
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Figure  8.  Percentage  of  Cracks  Detected  -  SHM  from  Time  Zero  Approach 


3.4.4  Risk-Based  for  The  First  Inspection  Approach  Results: 

This  approach  is  a  combination  of  approaches  2  and  3.  It  is  important  to  reduce  the  total  number 
of  inspections  and  repairs  based  on  SHM  approach.  Based  on  approach  2’s  first  inspection  time 
(determined  by  risk  of  1  .E-7),  it  will  greatly  reduce  the  number  of  initial  SHM  inspections 
because  the  initial  phase  of  crack  growth  will  not  have  a  lot  of  detections  and  repairs.  After  the 
first  inspection,  the  remaining  inspections  are  based  on  a  pre-selected  SHM  inspection  interval. 
Below,  both  the  single  flight  probability  of  failure  and  the  percentage  of  crack  detected  are  plotted 
in  Figure  9  and  10,  respectively. 


Hours 

Figure  9.  Single  Flight  Probability  of  Failure  -  Risk-based  NDI  Approach 
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Figure  10.  Percentage  of  Cracks  Detected  -  Risk-based  NDI  Approach 

3.5  Compare  Results  From  Both  CBM+SI  and  Deterministic  Approaches 

Step  1 .  Summarize  the  data  and  compare  with  the  results  obtained  by  using  deterministic 
approach 

As  discussed  earlier,  the  proposed  CBM+SI  strategy  will  apply  a  Boolean  type  of  SHM  to 
perform  the  inspection  and  repair.  As  the  integration  of  the  necessary  code  and  the  analysis  of 
the  SHM  system  POD  are  still  in  work  for  the  purposes  of  this  analysis  it  was  assumed  that  the 
SHM  system  would  have  similar  capability  to  the  current  NDI  including  expected  POD.  Four 
options  have  been  run  and  shown  in  the  previous  subsection.  The  results  of  these  four  options 
are  compared  and  shown  in  Table  1. 

As  shown  in  Table  1,  all  the  key  data  have  been  summarized  for  comparison.  Considering  the 
total  number  of  inspections  required  to  reach  8815  flight  hours,  the  deterministic  based  approach 
(Scenario  #  1)  used  14  inspections  and  the  NDI-risk  based  approach  (Scenario  #2)  used  only  12. 
For  the  deterministic  approach,  at  4550  hours,  the  risk  already  exceeded  l.E-7  level  and  reached 
up  to  5.4E-6.  The  risk-based  approach  was  able  to  maintain  the  risk  below  1  .E-7.  From  this 
comparison,  it  shows  the  importance  of  quantitative  information.  For  the  deterministic  based 
approach,  even  with  the  safety  factor  of  2,  the  outcome  may  be  un-conservative,  i.e.,  the  SFPOF 
may  exceed  the  l.E-7  threshold. 
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Table  1.  Results  Comparison 


Deterministic 

Approach 

Scenario  #1 

NDI -Risk 

Based  Approach 

Scenario  #2 

SHM  -  Risk  Based 
Approach  1 

Scenario  #3 

SHM  -  Risk  Based 
Approach  II 

Scenario  #4 

Inspection 

interval 

Determined? 

Based  on  a  safety 
factor  of  2 

Based  on  risk 
requirement  of 
l.E-7 

Tight  with  routine 
insp.  schedule  for 
easy  access 

Based  on  risk 
requirement  of  l.E-7 
for  the  first  insp. 

Inspection 
interval  hours 

650 

3000,  720, 590, 
580,  540,  520, 
500,  490,  480, 
470,  465,  460 

400  for  all 

inspection  intervals 

3000  as  the  first  one 
and  then  400  for  all 
inspection  intervals 

No.  of  inspection 
intervals  to  reach 
-  8815  hours 

14 

12 

23 

16 

Risk  level  range 

At  9100  hrs,  max. 
risk  =  5.47E-6;  at 
4550  hrs,  the  risk 
>  l.E-7  already 

Maximum  risk  = 
l.E-7. 

Very  small  risk  from 
the  beginning  to  the 
6.7E-9  at  8000  hrs.*** 

Max  risk  =  l.E-7  at 
3000  hours  and  then 
reduced.  At  9000, 
risk  =l.E-8.*** 

%  crack  detected 
&  repaired 

At  9100  hours, 
0.886%  detected. 

At  8815  hours, 
0.616%  detected. 

At  8000  hours, 
0.43%  detected.*** 

At  9000  hours, 
0.572%  detected.*** 

The  SHM  approach  from  time  zero  (Scenario  #3)  used  23  inspections  to  reach  8815  hours  and  the 
risk  stayed  below  1  .E-7.  The  highest  risk  level  was  found  to  be  around  6.7E-9.  For  the  last 
approach,  by  using  the  risk-based  approach  to  be  the  first  SFIM  inspection  time,  the  total  number  of 
inspections  was  reduced  from  23  to  16  and  the  risk  level  increased  to  l.E-8,  which  is  still  below 
the  1  .E-7  requirement.  These  two  SHM  related  approaches  definitely  need  to  perfonn  more 
inspections  than  the  risk-based  NDI  approach  but  the  risk  level  remains  much  smaller  than  the  risk- 
based  approach. 

In  addition  to  the  number  of  inspections,  the  other  key  data  to  be  recognized  is  the  %  of  cracks 
detected  and  repaired.  The  risk-based  approach  uses  fewer  inspections,  12,  in  comparison  to  the 
SHM  approaches  which  uses  23  and  16  inspections,  respectively.  The  risk  level  of  SHM 
approaches  remain  at  a  lower  level  than  the  risk-based  approach,  i.e.,  with  each  inspection  the 
crack  size  distribution  is  updated  after  an  inspection  with  new  information  that  either  no  cracks 
were  found,  or  cracks  were  detected  and  repaired  so  that  a  portion  of  the  distribution  was  updated 
based  on  the  repaired  crack  size  distribution. 

Based  on  the  above  data,  it  is  very  hard  to  detennine  which  approach  will  be  the  most  cost 
effective  maintenance  strategy.  To  perform  the  trade-off,  a  Technical  Performance  Measurement 
(TPM)  model  will  be  created  to  link  with  the  above  data.  Then,  the  developed  TPM  model  will  be 
used  to  estimate  the  overall  cost  for  each  approach  and  the  most  cost  effective,  technically 
acceptable  maintenance  option  identified.  Additional  discussion  of  the  proposed  TPM  model  can 
be  seen  in  the  TPM  Measurement  for  Success  section  of  this  report. 
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Step  2.  Identify  potential  shortcomings  and  gaps  to  be  improved  especially  when  applying  the 
developed  CBM+SI  strategy  to  the  fleet  level  (work  with  subtask  1 . 1 .2.6  “Forward  Looking 
Assessment”). 

Based  on  the  proposed  CBM+SI  strategy  and  the  demonstration  example,  potential 
shortcomings  and  gaps  will  be  identified  and  summarized  for  further  investigation  and 
improvement. 


20 

Approved  for  public  release;  distribution  unlimited. 


4.0  Sensitivity  Analysis 

The  objective  of  this  task  is  to  perform  a  series  of  sensitivity  analyses  to  determine,  which 
variables  contribute  the  largest  influence  on  the  key  technical  performance  measures.  In  addition 
an  optimization  approach  will  be  developed  which  demonstrates  how,  given  the  definition  of  the 
key  variables  involved,  an  optimum  maintenance  plan  can  be  created  which  minimizes  cost  and 
maximizes  aircraft  availability  while  maintaining  acceptable  risk  levels.  This  task  is  schedule  to 
start  in  August  and  be  completed  in  September. 
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5.0  Measurement  of  Success  via  Technical  Performance  Measurements 
(TPM) 


The  objective  of  this  task  is  to  conduct  the  business  case  analysis  on  implementing  CBM+SI 
solution  for  the  demonstration  system  (Task  1.1. 2.5)  and  show  the  impacts  for  the  aircraft  fleet. 
The  business  case  will  show  the  configurations’  impacts  on  the  fleet  availability,  labor  hour 
usage,  and  total  cost  of  ownership. 

5. 1  Establish  the  Baseline 

Task  1.1. 2. 5.1  in  the  Statement  of  Work  states,  “Boeing  shall  populate  the  operational  and  cost 
benefit  analysis  models  with  the  baseline  system’s  utilization,  reliability  &  maintainability,  and 
cost  data  in  order  to  define  the  baseline  system  for  comparison  to  the  CBM+SI  solution’s 
demonstrated  benefits.” 

Historic  maintainability  and  reliability  data  for  the  F-15C/D,  covering  operations  from  February 
2004  to  January  2009  were  gathered  from  the  Air  Force  Maintenance  Operational  Query  System 
(MOQS)  database.  This  historic  reference  was  used  to  establish  the  baseline  performance  of  the 
fleet.  A  key  metric  found  in  the  database  was  the  fleet’s  mission  capable  rate.  For  the  purpose 
of  this  study,  this  mission  capable  metric  was  used  as  the  baseline  fleet  availability,  for  the 
platforms  will  only  be  available  when  not  in  any  kind  of  maintenance.  This  value  was  used  to 
eventually  calculate  the  elapsed  downtime  for  all  maintenance  and  to  determine  that  portion  for 
the  lower  bulkhead. 

Although  maintenance  actions  were  recorded  for  the  FS  626.9  bulkhead  through  Work  Unit 
Code  1 1DAP,  none  of  them  had  records  covering  the  repairs  on  severe  cracks  in  this  bulkhead. 
Only  two  known  cracks  on  this  bulkhead  occurred  in  the  1980s  and  1990s,  but  data  in  the  Air 
Force  MOQS  only  cover  from  2000  to  2009.  Still,  Boeing  personnel  provided  insight  on  the 
occurrences  and  cost  of  these  failures,  and  the  maintenance  actions  and  their  labor  hours 
covering  this  Work  Unit  Code  were  noted.  A  possibility  may  exist  for  a  combined,  reduced 
duration  of  all  bulkhead  repairs  if  their  conditions  are  frequently  monitored. 

Also,  data  on  the  probabilities  of  bulkhead  failure  and  detection  of  cracks  were  provided  from  the 
previously  discussed  risk  analysis.  Analyses  were  conducted  on  two  F-15  Bulkhead  critical 
locations:  12B  and  237.  At  the  time  of  this  progress  report,  only  the  12B  zone  was  considered  in 
the  TPM  model. 

Using  the  historic  reliability  and  maintainability  data  and  the  risk  analysis,  a  baseline  model  was 
established.  Models  such  as  the  System  Health  Operations  Analysis  Model  (SHOAM)  and  the 
Probabilistic  IVHM  Cost-benefit  Analysis  Model  (PICAM)  were  considered  for  TPM  analysis; 
however,  these  modeling  tools  are  better  suited  to  analyze  a  series  of  implemented  solutions. 
Because  the  focus  is  only  on  the  lower  bulkhead,  a  simpler  model  was  created  to  calculate  the 
TPMs.  Although  further  analysis  on  the  repairs  for  zones  12B  and  237  were  provided  in  an 
earlier  section,  only  portions  of  that  analysis  became  the  basis  for  the  model. 

The  following  parameters  and  assumptions  were  established  for  the  baseline  model: 

•  Timeframe  for  the  model:  25  years 

•  Number  of  F-15  C/D  in  the  fleet:  300 
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•  Average  flight  hours  per  flight  (based  on  the  composite):  1 .3  Hrs 

•  Average  flight  hours  /  plane  /  year:  275  Hrs 

•  Number  of  holes  for  12B  on  the  platform:  18 

•  Labor  rate:  $80/Hour 

o  Estimated  rate  for  any  officer,  calculated  from  AFI  65-503,  Table  A32-1,  FY09 

•  Inflation  rate:  3% 

•  Discount  rate:  10.5% 

•  Unit  platfonn  cost:  $25M 

o  Cost  for  F-15E  is  approximately  $59M  from  Janes.com,  so  a  smaller  value  was 
used  for  the  F- 15 C/D. 

o  This  cost  is  used  to  calculate  the  monetary  equivalent  for  fleet  availability  change. 

•  Repairs  on  the  bulkhead  occur  right  after  scheduled  inspections. 

•  Time  spent  for  repairs:  1 .5  hours 

•  Cost  for  inspections  and  repairs  are  based  solely  on  the  labor  hours. 

•  Failure  cost  has  two  possibilities: 

o  Unit  platfonn  cost  ($25M)  to  represent  platform  loss,  or 

o  Estimated  cost  for  bulkhead  repair  based  on  past  experience  ($  1M),  which  includes 
approximately  1500  labor  hours  and  material  costs. 

5.2  Capture  Solution  Implementation  Costs 

From  Task  1.1. 2. 5. 2  in  the  Statement  of  Work,  “Boeing  shall  capture  the  estimated  development, 
production,  and  operating  &  support  costs  of  the  demonstration  system  in  the  cost  benefit  analysis 
model.” 

Technologies  similar  to  the  proposed  CBM+SI  alternatives  were  researched  to  estimate  the 
resources  and  time  needed  for  development,  production,  and  installation  on  target  platforms.  A 
similar  analysis  on  SHM  benefits  for  the  F-22  lug  was  conducted  within  the  company,  and  a 
model  from  a  fellow  Boeing  employee  contained  the  estimated  costs  to  develop  and  implement 
sensors  onto  a  structure. 

Below  are  the  estimated  costs  in  fiscal  year  (FY)  2009  dollars  for  developing  and  implementing 
a  set  of  sensors  to  the  12B  zone: 

•  Design  Cost  /  Unit:  $4,000 

•  Production  Cost  /  Unit:  $1,100 

•  Certification  Cost  /  Unit:  $4,000 

•  Physical  Unit  Cost:  $500 

•  Installation  Support  Material  Cost  /  Unit:  $2,000 

•  Installation  Labor  Cost  /  Unit:  5  hours  x  Labor  Rate 

To  cover  the  development,  production,  and  installation  of  SHM  solution,  the  following 
assumptions  were  established: 

•  Development  and  Certification  of  Solution:  Years  0  (Current)  and  1 

•  Production  and  Installation  of  Solution:  Years  2  and  3 

These  costs  for  SHM  approaches  were  entered  to  the  TPM  model,  and  the  total  costs  of 
ownership  between  the  baseline  and  modified  configurations  were  compared. 
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5.3  Capture  Solution  Benefits 

Task  1.1. 2. 5. 3  in  the  Statement  of  Work  states,  “Boeing  shall  capture  the  demonstrated  impact  of 
CBM+SI  on  the  system’s  reliability  &  maintainability  metrics  in  the  operational  and  cost  benefit 
analysis  models.” 

Four  scenarios  were  analyzed  to  determine  the  operational  and  cost  impacts  in  different 
configurations: 

•  #1 :  Detenninistic  (Baseline)  Approach  -  Fixed  inspection  interval  of  650  hours 

•  #2:  NDI  Risk  Based  Approach  -  Identify  inspection  intervals  based  on  the  risk 
threshold  of  10'7 

•  #3:  SHM  Approach  I  -  Identify  first  inspection  interval  based  on  the  risk  threshold  of 
1CT7,  and  then  use  fixed  inspection  interval  of  400  flight  hours 

•  #4:  SHM  Approach  II  -  Fixed  inspection  interval  of  400  flight  hours 

The  single  flight  probability  of  failure  and  the  probability  of  crack  detection  for  each  of  these 
scenarios  are  discussed  in  a  previous  section.  Note  that  the  risk  probabilities  after  each  flight  for 
scenarios  2-4  are  still  significantly  lower  than  the  baseline  approach.  Also  note  that  the 
implementation  of  the  SHM  solution  will  reduce  the  inspection  time  from  1 .5  hours  to  0.5  hours. 

Only  the  SHM  approaches  are  assumed  to  develop  and  implement  technology  to  the  platforms  and 
ground  operations.  Scenario  #2  was  assumed  to  not  add  additional  technology  onto  the  platforms, 
for  it  simply  meant  identifying  the  next  flight  hour  when  the  overall  risk  reached  10’7.  Meanwhile, 
the  intervals  are  set  in  the  SHM  approaches,  but  gathering  the  data  through  on-board  sensors  and 
conducting  analysis  can  reduce  the  risks  of  structural  failure. 

5.4  Determine  Operational  Benefit  Analysis 

Task  1.1. 2. 5. 4  from  the  Statement  of  Work  states,  “Boeing  shall  utilize  the  discrete- event 
simulation  operations  analysis  model  to  evaluate  the  baseline  and  CBM+SI  enhanced  system’s 
operational  metrics,  i.e.  availability.  The  delta  operational  benefit  for  CBM+SI  shall  be 
detennined  from  this  analysis.” 

The  simple  TPM  model  generated  the  fleet  operational  results  for  each  scenario,  shown  in 
Table  2. 

As  expected,  simply  adding  the  SHM  solution  to  cover  only  one  structural  area  has  minimal 
impact  on  the  fleet’s  mission  capable  rate,  but  the  downtimes  for  inspections  and  repairs  and  the 
number  of  expected  failures  are  smaller  than  the  deterministic  approach.  The  failure  quantity  is 
based  on  summing  all  of  the  expected  failures  for  each  flight. 
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Table  2.  Comparison  of  Performance  Results  for  Each  Scenario 


Deterministic 
Approach 
Scenario  #1 

NDI  Risk 
Based 
Approach 
Scenario  #2 

SHM 

Approach  1 
Scenario  #3 

SHM 

Approach  II 
Scenario  #4 

Total  Labor  Hours 

9089 

6390 

3103 

4166 

SHM  Installation 

1500 

1500 

Inspection 

9000 

6300 

1500 

2550 

Repair 

89 

90 

103 

116 

Change  in  Mission  Capable 

Rate 

baseline 

0% 

+0.01  % 

+0.01  % 

Total  Downtime  (Hours) 

9089 

6390 

1603 

2666 

Inspection 

9000 

6300 

1500 

2550 

Repair 

89 

90 

103 

116 

Expected  Number  of  Repairs 

59.2 

59.8 

68.9 

77.5 

Expected  Number  of  Failures 

0.318 

0.096 

0.042 

0.001 

5.5  Perform  Cost  Benefit  Analysis 

From  Task  1.1. 2. 5. 5  in  the  Statement  of  Work,  “Boeing  shall  evaluate  the  cost  benefit  of  the 
CBM+SI  solution  by  using  the  operational  benefit  analysis  results,  in  addition  to  the  solution 
costs  and  benefits,  captured  in  previous  steps  in  the  cost  benefit  analysis  model.  The  results  shall 
include  the  Net  Present  Value,  Return  on  Investment,  cash  flow,  and  maintenance  man-hours  per 
flight  hour.” 

The  simple  TPM  model  generated  the  fleet  monetary  results  for  each  scenario  in  Table  3. 

Table  3.  Comparison  of  Cost  Results  for  Each  Scenario 


Deterministic 
Approach 
Scenario  #1 

NDI  Risk 
Based 
Approach 
Scenario  #2 

SHM 

Approach  1 
Scenario  #3 

SHM 

Approach  II 
Scenario  #4 

SHM  Prod.  And  Install  Costs  ($) 

$  3,728,176 

$  3,728,176 

Expected  Inspection  Costs  ($) 

$  1,078,761 

$  852,576 

$202,561 

$  307,095 

Expected  Repair  Costs  ($) 

$  12,996 

$  13,047 

$  15,055 

$  17,170 

Expected  Fleet  Equiv.  Benefit  ($) 

$  1,313 

$  3,642 

$3,125 

Expected  Failure  Costs  ($) 

(When  Failure  Cost  =  $25M) 

$  11,747,761 

$  2,783,460 

$  1,023,712 

$  22,952 

Expected  Failure  Costs  ($) 

(When  Failure  Cost  =  $1 M) 

$469,910 

$  111,338 

$  40,948 

$918 
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The  formula  for  total  cost  of  ownership  when  focusing  on  the  lower  bulkhead 
maintenance  consists  of  the  following: 

Total  Cost  =  [SHM  Production  and  Installation  Costs] 

+  [Expected  Inspection  Costs] 

+  [Expected  Repair  Costs] 

+  [Expected  Failure  Costs] 

-  [Expected  Fleet  Equivalent  Benefit] 

While  implementing  one  of  the  SHM  approaches  may  show  overall  loss  compared  to  the 
baseline,  all  scenarios  do  not  yet  account  for  additional  material  costs  covering  repairs  and 
failures.  Also,  the  maturity  of  the  proposed  technology  can  affect  the  costs  to  develop, 
certify,  produce,  and  install  the  solution  to  the  fleet. 

5.6  Next  Steps 

The  TPM  model  will  be  adjusted  to  capture  the  costs  for  solution  development  and 
implementation,  depending  on  the  proposed  technology’s  maturity.  This  model  will  also 
include  the  costs  and  benefits  from  inspecting  additional  zones  such  as  Zone  237.  As  the 
baseline  and  proposed  configurations  are  refined,  the  range  of  costs  and  benefits  will  be  added 
to  the  model  to  detennine  the  likely,  minimum,  and  maximum  impacts  that  can  occur  for  the 
fleet. 
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6.0  Forward  Looking  Assessment 

The  objective  of  this  task  is  to  apply  simple  statistical  and  probabilistic  analysis  techniques  to 
the  methods  and  results  obtained  in  the  Demonstration  in  order  to  give  some  insight  into  how 
they  may  be  used  in  the  context  of  the  follow-on  of  Phase  II  project  as  applied  in  the  context  of 
the  aircraft  system,  and  fleet.  This  task  is  scheduled  to  start  in  August  and  be  completed  in 
September. 
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Schedule 
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1.1.2 
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1. 1.2.1  ASIP  integration 
1.1.22  SHMdata  integration 

1 .1.2.3  Data  analysis  &  prediction 

1.1.24  Se  ns  if  vity/uncertainty  analysis 

1 .1.25  Measurement  of  success  via  TPM 
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CDRL  Deliverables 
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AL 
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Appendix  A.  Input  Random  Variables  Data  Summary 


The  current  ASIP  is  ready  to  provide  the  basic  required  information  to  perforin  an  effective 
risk  assessment  except  for  a  probabilistic  analysis  strategy.  These  basic  requirements  are: 

•  Aircraft  usage  characterization:  Loads  and  Environmental  Spectra  Survey  (L/ESS)  or 
Individual  Aircraft  Tracking  (IAT)  data 

•  Crack  growth  and  residual  strength  based  on  demonstrated  usage  and  location  of 
interest,  material  parameters,  and  stress  intensity  solution  (a  =  K/o) 

•  Inspection  data:  Crack  size  and  usage  hours  at  detection 

With  this  data,  by  using  proper  uncertainties  modeling  strategy  to  convert  the  data  into 
statistical  distributions,  a  probabilistic  risk  assessment  can  be  used  to  compute  the  risk  and 
compare  with  the  single  flight  probability  of  failure  requirement  stated  in  Mil-Std-1530C 
(Reference  1).  With  the  proposed  probabilistic  risk  assessment  strategy,  the  safety  factor 
associated  with  the  existing  deterministic  approach  can  be  “converted”  into  the  risk  and  thus 
calibrated  accordingly. 

Based  on  the  proposed  risk  assessment  process,  strategies  for  uncertainty  modeling  play  an 
important  role  in  estimating  risk.  Fracture  toughness,  maximum  stress  during  a  flight,  and 
initial  crack  size  are  key  contributors  to  the  risk  estimation.  In  addition,  the  parameters 
required  for  the  NDI  POD  curve  and  repair  crack  size  distribution  are  key  contributors  for  risk 
mitigation  purpose. 

In  the  following  subsections,  a  list  of  random  variables  required  for  the  F-15  Bulkhead 
component  (12B  location)  risk  assessment  is  developed. 

•  Initial  crack  size  distribution 

•  Repair  crack  size  distribution 

•  Fracture  toughness  distribution 

•  Maximum  stress  distribution 

•  POD  parameters 

A.1  Initial  Crack  Size  Distribution 

For  the  F-15  Bulkhead  component  (12B  location),  the  initial  crack  size  distribution  selected 
for  the  Titanium  material  Ti-6A1-4V  is  a  Weibull  distribution  with  Shape  parameter  and 
Scale  parameter  equal  to  1.541  and  0.0022,  respectively.  This  distribution  was  recommended 
by  Mr.  Chuck  Babish,  Air  Force  Life  Cycle  Management  Center  Technical  Advisor,  Aircraft 
Structural  Integrity. 

A. 2  Repair  Crack  Size  Distribution 

In  general,  the  repair  crack  size  distribution  was  considered  equal  to  the  initial  crack  size 
distribution  because  it  is  assumed  the  repair  quality  is  good  so  the  quality  of  the  crack  size 
distribution  should  be  as  good  as  new. 
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A. 3  Fracture  Toughness  (Kc)  Normal  Distribution 

Without  actual  data,  normal  distributions  with  the  coefficient  of  variation  (p/o)  from  about 
3%  to  10%  for  aluminum  and  titanium  alloys  and  most  steels  are  used  to  model  the  Kc 
distribution.  For  this  Titanium  material  Ti-6A1-4V,  the  mean  value  equals  to  100.2  and  the 
standard  deviation  equal  to  10  were  assumed,  i.e.,  a  10%  COV  was  assumed  for 
conservatism. 

A.4  Maximum  Stress  Gumbel  Distribution 

As  discussed  in  the  input  data  requirement  subsection,  the  methodology  to  determine  the 
distribution  of  the  maximum  stress  Gumbel  distribution  had  been  discussed  in  details.  Based 
on  the  procedure  proposed  by  the  PROF  code,  the  maximum  stress  Gumbel  distribution  was 
calculated  using  the  load  exceedance  data  and  a  least  square  fit  was  used  to  detennine  the 
Gumble  EVD.  As  shown  in  Figure  A-l,  three  different  least  square  fit  curves  were  plotted  for 
comparison.  For  conservatism,  an  8-points  fit  (with  Scale  (Asig)  =  0.0468  and  Location 
(Bsig)  =  0.97)  was  selected  for  the  Gumble  EVD. 
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Figure  A-l.  Least  Squares  Fit  to  Stress  Exceedances 


A. 5  Probability  Of  Detection  (POD)  Parameters 

To  model  the  probabilities  of  detection  (POD)  curves,  the  crack  size  that  can  be  detected  with 
90%  detection  and  95%  confidence  level  must  first  be  identified.  For  the  12B  location,  a  bolt 
hole  eddy  current  inspection  method  will  be  used  to  inspect  the  component.  Based  on  this 
method,  the  90%  detection  and  95%  confidence  level  was  found  equal  to  0.05  inches.  In 
addition  to  this  value,  the  median  value  with  a  50%  detection  and  95%  confidence  level  was 
found  at  0.025  inches.  With  these  two  values,  it  is  possible  to  establish  a  lognormal  POD 
model,  Figure  A-2,  with  the  following  parameters: 
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•  amed  (median  value  of  X)  =  0.025 

•  asteep  =  0.5 

•  amin  (minimum  value  of  X)  =  0.003  inches 

•  and  Probability  Of  Inspection  (POI)  =  1 


Crack  Size  (Inches) 

Figure  A-2.  Probability  of  Detection  Curve 


A. 6  Important  Deterministic  Input  Data 

Crack  Growth  Curve  (Crack  Size  a  (inches)  vs.  Flight  Hours  T)  -  the  same  deterministic 
Damage  Tolerance  Analysis  (DTA)  results  will  be  used  to  perform  the  growth  of  fatigue 
cracks  by  projecting  the  percentiles  of  the  fatigue  crack  size  distribution. 

Geometry  (Crack  Size  a  (inches)  vs.  K/sigma)  -  Under  current  Air  Force  regulations,  damage 
tolerance  analyses  are  performed  for  every  critical  location  on  an  airframe.  As  part  of  these 
analyses,  the  stress  intensity  factor  geometry  correction,  P(a)  ,  for  correlating  stress,  loading 
condition,  global  geometry,  and  crack  size  will  have  been  determined.  The  geometry  file 
also  plays  an  important  role  when  computing  the  risk  as  shown  in  the  following  failure  mode 
or  limit  state  function: 


a  = 


& max 
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The  other  key  failure  mode  occurs  when  the  crack  grows  to  exceed  the  critical  crack 
length  as  shown  in  the  following  limit  state  function: 

§  &  @cr 

For  interval  probability  of  failure,  the  following  data  are  important: 

•  Similar  locations  in  an  aircraft  =  28 

•  No.  of  aircraft  in  a  fleet  =  300 

•  Average  hours  per  flight  =1.3  hours. 
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